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This study examines administrators’ perspectives of a state-developed principal evaluation 
model adopted by a majority of Indiana school districts after legislation mandated policy reform 
in educator evaluation. Feedback was gathered from public school superintendents (the 
evaluators) and principals (those being evaluated), with 364 participants. Mixed-methods 
analyses revealed areas of model utility, implementation challenges, and statistically significant 
differences between principals ’ and superintendents ’ perceptions of model efficacy. Both 
superintendents and principals agreed the new model was an improvement over traditional, 
locally developed models; however, superintendents rated the model higher in efficacy and 
implementation fidelity. Principal participants did not perceive the new model to be effective for 
principal evaluation and would not recommend the model to colleagues. Implications for future 
policy development in educator evaluation include the integration of findings from essential 
evaluative research. Recommendations for practice include suggestions for facilitating field- 
based support when developing and implementing evaluative models. 
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The principal’s role is complex and multi-faceted. Research and federal mandates for 
accountability have attempted to define, categorize, and connect human behavior to the job in 
order to determine the most effective practices that promote student achievement (Marzano, 
Waters, & McNulty, 2005; Stronge, 2013). However, the link between principal evaluation and 
effective leadership for evaluative purposes continues to elude the education field (Fuller & 
Hollingsworth, 2013). The need for deeper understanding is underscored by the lack of 
empirical evidence to connect evaluation to effective leadership while researchers continue to 
search for scientifically proven practices (Shelton, 2013). 

History is replete with educational change efforts and the American educational system has 
experienced many reforms. Recent reforms have resulted in an increased emphasis on the 
evaluation of educators due to the ideology that schools, as places for reshaping individuals and 
reforming society, must improve and educators are significant in this process (Tyack & Cuban, 
1995). Early in the twenty-first century, the No Child Left Behind Act required states to 
dramatically increase educator evaluative and qualification standards, introducing “highly 
qualified” accountability (2001). In 2009, the Race to the Top program required additional 
evaluative accountability in order for school districts to be eligible for substantial federal grants 
(Manna & Ryan, 2011). These educational refonn efforts had noteworthy influences on the 
direction of educational policy regarding the evaluation of educators (Hazi & Rucinski, 2009). 

Traditional principal evaluation models have been called into question in terms of fidelity of 
implementation and actual impacts on principal effectiveness (Davis & Hensley, 1999; New 
Leaders for New Schools, 2010; Reeves, 2006). These models were typically summative in 
nature and characterized by one or two (or sometimes zero) school observations conducted by 
the principal’s supervisor (usually the superintendent or the superintendent’s designee), followed 
by a written year-end evaluation (Portin, Feldman, & Knapp, 2006). 

In 2006, Reeves found that traditional principal evaluations were often based on brief annual 
observations, and grounded in little or no context, hearsay, and exaggerated one-time situations. 
In Reeve’s study, 60% of principals felt their evaluations had no impact on their job 
performance. In 2000, Thomas, Holdaway, and Ward reported that supervisors responsible for 
evaluating principals inconsistently conducted such evaluations, concluding that principal 
performance was inconsistently measured. Thomas et al. also suggested that there were 
differences between how principals and superintendents viewed principal evaluation in terms of 
importance and usefulness (2000). 

Davis and Hensley (1999) found that principals regarded their evaluations as something that 
happened to them, not something that was useful for improving job performance. In addition, 
Davis and Hensley noted that principals’ felt their evaluations were influenced by external 
political factors, such as parents and board members, rather than daily practice. Other reports 
have suggested that traditional evaluations served more to maintain the status quo than to 
promote educator effectiveness (New Leaders for New Schools, 2010). 

Meanwhile, a body of research developed showing that the effectiveness of the principal 
made a significant difference for student academic achievement and overall school success 
(Branch, Hanushek, & Rivkin, 2013; Leithwood, Louis, Anderson, & Wahlstrom, 2004; 
Marzano et al., 2005; King-Rice, 2010). In a landmark report from The Wallace Foundation, 
principals were found to be second, only behind classroom teachers, in terms of school-related 
factors influencing student learning (Leithwood et al., 2004). Leithwood et al. provided three 
sets of practices that were basic to successful leadership: Setting direction, developing people, 
and redesigning the organization. The practice of setting directions implies that the leader 
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develops shared meaning in the school, which will lead to a strong purpose or vision with 
challenging, but achievable goals. The second set of practices centers on developing people. In 
order to build the capacity of the people within the organization, effective leaders stimulate 
thinking, support individual needs, and model best practices and values. When redesigning an 
organization, leaders strengthen the culture, build collaboration, and modify the structures of the 
organization so that the organization adapts with the changes in the school improvement process. 
Leithwood et al. (2004) also reported that the impact of leadership was most significant in the 
neediest of schools, emphasizing the importance of highly-effective leadership in schools 
considered “failing.” 

In 2005, Marzano et al. conducted a meta-analysis of 69 studies, resulting in a list of 21 
responsibilities of effective school leaders. This study was considered a milestone in the attempt 
to identify leadership behaviors that lead to improved student achievement. These 21 
responsibilities included important practices and attributes demonstrating principals’ leadership 
effectiveness in areas such as communication, knowledge of curriculum, instruction, and 
assessment, and the ability to be a change agent. 

In 2011, the National Association of Elementary School Principals (NAESP) and the 
National Association of Secondary School Principals (NASSP) joined forces to form a principal 
evaluation committee with the objective of creating a research-based framework for evaluation 
designed to build principals’ leadership capacities. Through the committee’s work, the following 
seven essential features of quality evaluation were identified: Systemic support, utility, 
flexibility, accuracy (validity and reliability), relevance, fairness, and creation by and for 
principals (Clifford & Ross, 2011). Then, in 2013, Stronge, Xu, Leeper, and Tonneson outlined 
a summary of findings on qualities of effective principals based on notable research in the field. 
These reports, as well as others, made strong contributions towards reform efforts in principal 
evaluation practices. 

Due to increased attention on the significance of effective school leadership, in concert with 
heightened accountability, policy makers began to view the principal as a key educational 
variable (Hallinger & Heck, 1996). In recent years, there has been considerable state-level 
momentum towards the development of new principal evaluation models that not only measure 
performance for accountability purposes, but also promote improvement efforts, requiring both 
formative and summative evaluations (Jacques, Clifford, & Hornung, 2012; Samuels, 2011). 
Since 2005, thirty-four states have passed legislation requiring district adoption of new principal 
evaluation systems (Jacques et al., 2012). A number of states now require value-added models 
(VAMs) that use student academic assessment data as one factor in measurement of principals’ 
effectiveness ratings (American Educational Research Association [AERA], 2015; American 
Statistical Association [ASA], 2014; Samuels, 2011). 

However, even as new evaluative models are designed and introduced, school leadership 
paradigms are shifting and expectations for principals growing. American society is becoming 
increasingly complex, diverse, globally networked, and technologically driven. School leaders 
must understand and address much more than student achievement, but must also address 
important concerns impacting school-community culture, such as safety, equity, diversity, and 
social justice (Kemp-Graham, 2015; Miller & Martin, 2015). The challenges of school 
leadership in an increasingly complex and diverse society, combined with growing demands for 
accountability and political scrutiny to increase student achievement, validates the need to 
understand the models and practices that are being utilized to evaluate school leaders. 
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Indiana Context 


Historically, Indiana has allowed its 290 public school districts a great deal of local autonomy in 
the development and implementation of evaluation policies for educators. However, Indiana’s 
Public Law 90 (PL 90) passed in 2011, mandated substantial change, calling for refonn in 
educator evaluation. This law required that every Indiana school district develop and implement 
a state-approved evaluation model for teachers and administrators. Although districts were still 
allowed to develop their own models, the implementation timeline was tight and all components 
outlined in the legislation were compulsory. Requirements included annual performance 
evaluations for all educators using effectiveness ratings of one (ineffective) to four (highly 
effective). The mandated effectiveness ratings were to be based, in part, on student achievement 
indicators (test scores) and ratings in six identified “competencies” of effectiveness. These 
competencies were aligned with the Indiana Content Standards for Building-Level Leaders and 
included: Instructional Leadership, Building Relationships, Student Learning, Human Capital 
Management, Culture of Achievement, and Personal Behavior (Indiana Department of Education 
[IDOE], 2010). Significant consequences were outlined for educators who scored in the two 
lowest evaluation categories (ineffective or improvement needed), including being blocked from 
receiving raises and being dismissed (IDOE, 2012a). 

In 2011, the IDOE developed a proposed model for teacher evaluation, which included a 
comprehensive evaluation rubric. Shortly after, in 2012, the IDOE put forth the model and 
rubric for principal evaluation. The teacher and principal evaluation models were tenned 
“RISE” (the RISE Evaluation and Development Systems). In developing the RISE Principal 
Evaluation and Development System, a team reviewed many nationally recognized publications 
in the area of principal leadership, including work by Doug Reeves, Todd Whittaker, VAL-ED, 
the National Board’s Accomplished Principal Standards, and several other models of effective 
evaluation. However, it is unclear from the documentation provided by the Indiana Department 
of Education how well the RISE model and rubric align with other research cited in this article or 
the research (Young & Mawhinney, 2012) supporting the ELCC standards. 

The RISE principal evaluation model requires annual protocol (multiple observations and 
conferences), and the IDOE provided training opportunities for evaluators throughout the state. 
Two major components, professional practice and student learning, provide the data sources to 
rate a principal’s perfonnance. Professional practice is measured by using the Indiana Principal 
Effectiveness Rubric. Two domains comprise the principal evaluation rubric: teacher 
effectiveness and leadership actions. To promote teacher effectiveness, principals must 
demonstrate competencies in human capital management, instructional leadership, and the ability 
to identify indicators of student learning in the school. Some of the more discrete 23 sub¬ 
competencies aligned with teacher effectiveness include areas such as hiring and retention, 
evaluation of teachers, professional development, and addressing teachers who are in need of 
improvement or ineffective. 

The Leadership Actions domain on the RISE principal evaluation rubric incorporates the 
three areas of 1) personal behavior (such as professionalism and using feedback effectively), 2) 
building relationships (such as forging consensus for change and communication), and 3) 
creating a culture of achievement. To create a culture of achievement, the rubric assesses sub¬ 
competencies like high expectations, academic rigor, and data usage in teams. 

In addition to the principal evaluation rubric, the second major component used to assess 
Indiana principals involves measurements of student learning. The state includes several 
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measures of student learning. Indiana annually rates schools with an A-F grade based on the 
student achievement results. That rating is added as a percentage into the overall evaluation of 
the principal. Furthermore, the principal sets two administrative student learning objectives, 
which are weighed equally and again added as a percentage into the final evaluation for the 
principal. These learning objectives can focus on growth or achievement goals. Finally, the 
summative score given to the principal is based on these metrics: Principal evaluation rubric at 
50%, the school’s A-F grade at 30%, and administrative student learning objectives at 20%. 

Several of the evaluative practices included in RISE and described above are considered 
“emergent,” such as the Administrative Student Learning Objectives (school-wide academic 
goals), rubric-based assessment, and VAMs that have been integrated into Indiana’s model for 
principal evaluation. Emergent practices are newer evaluative strategies; in some cases, 
considered best practice, such as rubric-based assessment (Danielson, 2011). However, some 
emergent practices, such as VAMs, are not yet fully vetted and considered problematic (AERA, 
2015; ASA, 2014; Darling-Hammond et ah, 2012; Newton et ah, 2010). 

The RISE model for teachers, including the evaluation rubric, was piloted in three Indiana 
school districts during the 2011-2012 school year. After the pilot year, the teacher evaluation 
model was revised slightly based on recommendations from the pilot participants. At that time, 
almost 80% of Indiana districts reported intent to implement the RISE models (IDOE, 2012b). 
However, no pilot for the RISE principal evaluation model ever took place. Indiana school 
districts needed to move quickly in order to be compliant with the new law and many districts 
adopted the RISE models as written by the IDOE in 2012. Other school districts adopted RISE, 
but made “allowable modifications” to the model. Since the development and implementation of 
the RISE principal evaluation model in 2012, no statewide studies had been conducted to 
examine practitioners’ perceptions of the model, prompting the need for this study. 

Purpose 

“While principal evaluation holds great potential, a relatively small number of studies on 
principal evaluation practices are available, and those suggest that improvements are long 
overdue. The studies raise questions about the consistency, fairness, effectiveness, 
accountability, and value of current principal evaluation practice” (Clifford & Ross, 2011, p. 2). 
School districts are moving away from traditional practices for the evaluation of public school 
educators, and there is momentum towards state-level reform, including the use of emergent 
evaluative practices (Gullickson, 2009; Jacques et ah, 2012; New Leaders for New Schools, 
2010). This may be because traditional principal evaluation has not been routine and systematic, 
and evaluations have not been comprehensive, informed by valid measures, or aligned with 
contemporary professional standards (Clifford & Ross, 2011; Davis, Kearney, & Sanders, 2011). 
In addition, principals have not viewed evaluation systems as providing valuable feedback to 
improve their practice (Reeves, 2006; Davis & Hensley, 1999). 

However, the landscape is changing. Indiana has aggressively moved forward with an 
innovative principal evaluation model with emergent practices such as rubric-based assessment, 
VAMs, required evaluator training, regular observations, and principal/evaluator conferences 
meant to encourage principals’ professional growth. The RISE principal evaluation model 
recommends five direct observations per year and three conferences. In contrast, PL 90 does not 
specify the number of observations, but requires a rigorous annual evaluation with objective 
measures used to inform the evaluation. The rigorous methods of the RISE program of 
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evaluation include student achievement results and other perfonnance indicators, but observation 
requirements are not designated. In sum, the law is vague about observation specifics, but the 
RISE model is clear in recommending multiple observations with prompt feedback. One of the 
intentions of the RISE model is to facilitate feedback to increase principal effectiveness. The 
purpose of this study was to gather and examine practitioners’ opinions as to whether the new 
model is meeting intended purposes. Seven research questions guided this study: 

1. To what extent do Indiana superintendents and principals perceive that the RISE principal 
evaluation model supports improvement of principal leadership? 

2. What competencies do Indiana’s superintendents and principals identify as most important for 
principals? Do participants’ rankings align with IDOE priorities for principal effectiveness? 

3. For districts implementing allowable modifications to RISE, what modifications are being made? 

4. What are Indiana’s superintendents’ and principals’ perceptions regarding the levels of fidelity of 
implementation of the RISE model? 

5. What are Indiana’s superintendents’ and principals’ ratings of effectiveness of the RISE rubric as 
a tool for evaluating principals? 

6. Would superintendents and principals recommend the RISE model to a colleague not currently 
using the model? 

7. What do participants perceive are the strengths and challenges of the RISE model? 

Methodology 

Our goal was to obtain information directly from Indiana principals and superintendents 
regarding their perceptions of the RISE model for principal evaluation. We also wanted to 
compare the responses of principals with those of superintendents in order to determine any 
differences in perceptions between those being evaluated and the evaluators. In Indiana, 
superintendents or their designees (e.g. assistant superintendents) are responsible for principal 
evaluation, and this duty is typically assigned to one person per school district. 

Based on our goals, an anonymous online survey approach was deemed most appropriate. 
Superintendents’ and principals’ whose school districts used the RISE principal evaluation 
model or a modified version were asked to participate. The survey first gathered participants’ 
basic demographic information, including age, gender, degrees earned, and years of experience. 
General information about the school district was also obtained, including student enrollment, 
percent of students on free/reduced meals, and school community type (rural, suburban, or 
urban). The rationale for collecting this demographic infonnation was to provide a context of 
those who participated in the study and determine if this sample was representative of the state. 

Next, using a Likert-type scale, the survey asked eleven “perception” questions designed to 
gather participants’ views on the RISE model. Several open-ended survey items were also 
included to allow respondents the opportunity to provide narrative regarding perceived strengths 
or challenges of the model. All survey items were directly related to the study’s research 
questions and grounded in the literature review. Before implementation, a panel experienced in 
survey development reviewed the entire survey and submitted feedback regarding face and 
content validity. This seven-member panel consisted of university faculty members and several 
recently retired Indiana principals and superintendents. Based on the panel’s feedback, several 
edits and wording revisions were made to improve survey clarity and flow. The survey was then 
pilot tested with a similar panel with favorable results. 
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Data Sources 


After pilot testing, the survey link was made available through the Indiana statewide associations 
for principals and superintendents (the Indiana Association of School Principals [IASP] and the 
Indiana Association of Public School Superintendents [IAPSS]). Thus, this study was an open 
call to any public school principal or superintendent through their membership associations. The 
survey was launched in early February 2015, with 364 respondents by the beginning of March 
2015, when the survey was officially closed. Based on the IASP and IAPSS membership 
numbers, we determined a response rate of approximately 22%. For the perception questions 
using a standardized Likert-type scale, the Cronbach alpha coefficient for internal consistency 
reliability was .924, establishing high reliability (Gay & Airasian, 2000). 

After compiling results, descriptive statistics were utilized to provide an overview of 
participants’ responses. Then, inferential analyses were conducted to compare principals’ and 
superintendents’ mean responses on the Likert-type scale questions. Finally, the narrative 
responses from the open-ended questions were coded and categorized into emerging themes 
(Saldana, 2009; Strauss & Corbin, 1990). 


Results 

Principals’ and superintendents’ mean responses for Likert-type scale items were analyzed, 
broken down by two subgroups, superintendents and principals, compared and examined 
quantitatively. Then, qualitative techniques were employed to scrutinize the narrative responses. 
In the following paragraphs, demographic responses will be summarized first, followed by an 
analysis of results in alignment with each research question. 

Demographics 

Demographic data collected on 364 respondents revealed that 68% (n = 231) were principals and 
32% (n = 108) superintendents. Of the principals, 58% (n = 135) indicated male and 42% (n = 
96) female. Of superintendents, 78% (n = 85) were male and 21% (n = 23) female. Overall, the 
participants described as 65% ( n = 220) male and 35% (n = 119) female. 

The typical principal was in the age range of 41-50 years (36%, n = 83), while 37% (n = 40) 
of superintendents were in the age range of 51-60. The majority of principals, 41% (n = 113) 
and superintendents, 41% (n = 38) reported having between four to ten years total experience in 
their positions. Most principals, 78% (n =181) had earned a Master’s degree, while 45% (n = 
48) of superintendents had earned an Education Specialist’s degree, and another 47% (n = 50) of 
superintendents had earned a Doctorate degree. 

A total of 40% (n = 134) of respondents, 58% (n = 89) principals, and 42% (n = 45) 
superintendents, served student populations with 41-60% qualifying for free or reduced meals. 
An additional 20% (n = 67) of principals and superintendents reported serving student 
populations with greater than 61% qualifying for free or reduced meals. Only 13% (n = 13) 
reported student enrollments of 20% or less qualifying for free or reduced meals. 

School community was defined as rural by 58% of participants, with suburban next at 25%, 
and urban at 17%. Overall, it was observed that the school communities of the participants 
presented an accurate representation of Indiana (U.S. Census Bureau, 2010). Indiana is primarily 
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a rural state, with only 16 areas considered “large urban” (Indiana State Government, 2009). 
There are a total of 1,933 public schools in the state serving 1,040,765 students (IDOE, 2015). 

Research Question One 

1. To what extent do Indiana superintendents and principals perceive that the RISE principal 
evaluation model supports improvement of principal leadership? 

Respondents were asked to rate the RISE model in terms of how well it supported 
improvement in principals’ overall leadership effectiveness using a Likert-type scale (1 = 
Strongly Agree to 4 = Strongly Disagree). When principals and superintendents were combined 
(n = 318), responses indicated agreement that the RISE model supported principals’ leadership 
effectiveness (M = 2.17, SD = .571). When the two groups were separated, superintendents (n = 
103) reported higher agreement (M = 2.01, SD = .495), and principals (n = 215) indicated a 
slightly lower level of agreement with a mean of 2.25 (SD = .590), and a statistically significant 
difference was revealed, t(316) = 3.592 ,p = .000. 

Research Question Two 

2. What competencies do participants identify as most important for principals? Do 
participants’ rankings align with IDOE standards-based priorities for principal effectiveness? 

The survey asked respondents to rank the six competencies in the RISE principal 
evaluation model in the order of importance with 1 = Most Important and 6 = Least Important. 
Not all participants selected a ranking for each competency and a few ranked more than one 
competency the same. Combining participants’ selections of 1, 2, or 3 assisted in determining 
the competencies deemed most important. It was clear from these combined rankings that 
Student Learning was viewed as the most important competency, 75% (n = 230). The second 
most important competency was Instructional Leadership, 67% (n = 207), followed by Building 
Relationships, 54% (n = 165) and Culture of Achievement, 50% (n = 155). Human Capital 
Management, 27% (n = 85) and Personal Behavior, 26% (n = 80) were noted as the least 
important competencies for principals. Table 1 summarizes responses. 

Table 1 


Rank of Competencies in Order of Importance as Reported by Superintendents and Principals 


Rank 

Competency 

n 

% 

M 

SD 

1. 

Student Learning 

230 

75 

2.59 

1.52 

2. 

Instructional Leadership 

207 

67 

2.84 

1.64 

3. 

Building Relationships 

165 

54 

3.30 

1.46 

4. 

Culture of Achievement 

155 

50 

3.54 

1.55 

5. 

Human Capital Management 

85 

27 

4.36 

1.55 

6. 

Personal Behavior 

80 

26 

4.35 

1.69 


Note. Selection of a 1, 2, or 3 is combined to indicate that Principals and Superintendents rate the competency as Most Important 


Next, the participants’ ranking of the competencies were compared to the ra nk or 
“weight” assigned to the competencies by the IDOE on the Principal Licensure Assessment 
(IDOE, 2012c) and outlined in the Indiana Content Standards for Building-Level Leaders (IDOE 
2010). By examining the assessment blueprint (IDOE, 2012c) and analyzing the weights given 
to each area, we determined that Human Capital Management was weighted as the most 
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important competency (25% weight). Next in weight were: Instructional Leadership, Student 
Learning and Culture of Achievement, Personal Behavior, Building Relationships, and 
Organizational (Operational and Resource) Management, with the last three standards receiving 
equal weight per the IDOE (2012c). In sum, there was a discrepancy between the state’s priority 
competency and the competency our participants perceived to be most important for principals. 

Research Question Three 

3. For districts implementing modifications to RISE, what modifications were being made? 

The RISE principal evaluation model is a statewide model introduced for use in school 
districts, but there were “allowable modifications” per the IDOE. Our survey was designed to 
discriminate if districts were using the model as it was originally developed or a modified 
version. If participants’ indicated their district had modified the model they were asked to 
describe the modifications that had been implemented. These open-ended responses were coded 
and then refined through a categorization process to identify themes (Saldana, 2009). 

Of the 305 respondents answering this question, 60% (n = 182) indicated their district used 
the RISE model as originally developed and 40% (n = 123) had made modifications. The 123 
participants who indicated a modified model were asked to describe the modifications made, and 
there were 78 useable responses. Of these, the most common modification described changing 
the language or wording of the rubric. For example, a participant stated, “The original wording 
is negative, our school district rewrote the rubric to demonstrate the behaviors and outcomes we 
want to observe. We combined some areas as it seemed redundant.” Another wrote, “The RISE 
system has been modified to lessen the harshness of the language of several of the indicators.” 

The second most common modification was changing the rubric metrics. An example 
comments was, “We don’t use all of the criteria; we have selected those that are the most 
important to us.” The next most common modification was changing the number of 
observations, followed by changing everything possible, and using only the rubric. As one 
respondent noted, “We have modified all of the RISE system as RISE was way over the top!” 
Multiple participants indicated that their districts had created different schedules, rubrics, 
timelines, and suggested protocols. In addition, several participants noted that the criteria for 
evaluations were not being followed in their districts. Table 2 summarizes the modification 
categories reported. 

Table 2 


Types of Modifications to the RISE Model as Reported by Participants 


Modifications 

n 

% 

Changed mbric 

27 

35 

Changed metrics 

17 

22 

Changed number observations 

13 

17 

Changed everything possible 

11 

14 

Criteria not being followed 

5 

6 

Use rubric only 

2 

3 

Other 

3 

4 

Total 

78 

100 
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Research Question Four 

4. What are Indiana’s superintendents’ and principals’ perceptions regarding the levels of fidelity of 
implementation of the RISE principal evaluation model? 

The next section of the survey collected responses on a Likert-type scale asking respondents 
to rate the RISE model as to fidelity of implementation according to the requirements. 
Combining principals’ and superintendents’ responses (n = 300), the mean suggested agreement 
that the RISE model was being implemented with fidelity (M = 2.23, SD = .783). However, 
when separating the two groups, superintendents (n = 99) reported higher agreement (M = 2.06, 
SD = .793), and principals (n = 201) indicated a lower level of agreement with a mean of 2.31 
(SD = .766). A statistically significant difference was found between the superintendents’ and 
principal’s mean responses (?(298) = 2.658, p = .008) for fidelity of implementation. 

Research Question Five 

5. What are Indiana’s superintendents’ and principals’ ratings of effectiveness of the RISE 
principal evaluation model as a tool for evaluating principals? 

Respondents were asked to rate the RISE model as to whether they perceived it to be an 
effective tool for evaluating principals. Together, principals’ and superintendents’ (n = 303) 
responses indicated agreement, suggesting the model was viewed as an effective tool for 
evaluating principals (M = 2.40, SD = .668). However, upon separating the groups, 
superintendents’ (n = 98) reported agreement (M = 2.17, SD = .658), while the principals’ mean 
indicated disagreement ( n = 205, M= 2.51, SD = .646), with a statistically significant difference 
found (7(301) = 4.242, p = .000). 

Research Question Six 

6. Would participants recommend the RISE model to a colleague not currently using the model? 

This question asked respondents if they would recommend the RISE model to their 
colleagues. Principals’ and superintendents’ combined mean response indicated they would not 
recommend the model to colleagues (M= 2.52, SD = .713). When separated, superintendents (n 
= 99) reported agreement (M = 2.34, SD = .717). However, principals (n = 204) reported 
disagreement with a mean of 2.61 (SD = .697), indicating they would not recommend the model. 
Once again, inferential analyses revealed a statistically significant difference between the 
superintendents’ and principals’ mean responses, ?(301) = 3.069, p = .002. 

Research Question Seven 

7. What do participants perceive are strengths and challenges of the RISE model? 

At the end of the survey, open-ended items allowed participants to provide written 
commentary about perceived strengths and challenges of the model. These items were: 
“Compared to the evaluation system my district previously used, describe the strengths of the 
RISE Principal Evaluation and Development System,” and “Compared to the evaluation system 
my district previously used, describe challenges of the RISE Principal Evaluation and 
Development System.” 
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For the question asking about the strengths of the model, 189 participants provided a total of 
195 responses, which were hand-coded. The theme of “clear expectations,” was most prominent, 
with participants noting that the RISE model had a clearer definition of principal expectations 
than previous evaluation tools or models. For example, one principal stated, “It is crystal clear 
on what needs to happen at each area for effective and highly effective performance.” 

In addition, the evaluation rubric was seen as a strength of the model, with comments like, 
“The RISE rubric hits the areas that are important in improving student achievement.” 
Participants also described increased communication between principals and supervisors as a 
strength, with one superintendent noting, “... it can be used as a collaborative tool on an on¬ 
going basis.” Other strengths mentioned by participants included a heightened focus on using 
data and the model prompting principals to spend more time with teachers. 

Although this question asked about strengths, multiple participants commented that they 
could find no strengths with the RISE model. One principal noted, “It doesn’t correlate to what 
is actually done in a building on a daily basis.” Another principal noted, “It could be beneficial 
if it was utilized effectively,” indicating a lack of fidelity in implementation. One superintendent 
commented, “RISE does not provide any benefit compared to our old system,” and another 
stated, “... it is somewhat better, but I do not believe it is a good evaluation tool.” Table 3 
summarizes the themes derived from comments regarding perceived strengths of the model. 

Table 3 

Descriptors of the Strengths of the RISE Principal Evaluation Model 


Principals Superintendents Total 


Strengths of RISE 

n 

% 

n 

% 

n 

% 

Clear expectations 

37 

30 

25 

35 

62 

32 

There are no strengths 

28 

23 

11 

15 

39 

20 

More focused on data 

19 

15 

12 

17 

31 

16 

Rubric is a strength 

14 

11 

10 

14 

24 

12 

Comments not useable 

14 

11 

4 

6 

18 

9 

More communication 

8 

6 

4 

6 

12 

6 

More time with teachers 

4 

3 

5 

7 

9 

5 

Total 

124 

100 

71 

100 

195 

100 


The next open-ended survey item asked about perceived challenges of the RISE model. 
There were 189 participants who responded, providing 227 comments. After these were coded 
and categorized, the model being “too time consuming,” was the challenge most commonly 
expressed by both superintendents and principals. When the two groups were separated, more 
superintendents than principals noted concerns about the time commitments required to conduct 
evaluations. Many superintendent comments had a negative tone such as, “Time and the 
observation of the principal on location seem, in many cases, rather contrived and less than 
productive.” Another said, “Time is always the biggest challenge as it has been difficult for me 
to arrange observations of principals.” A principal observed, “It is very time consuming. 
Superintendents do not have time to see us work in each of the evaluated areas.” 

Another perceived challenge was lack of fidelity in implementation. There were 
recurring comments from participants, especially principals, discussing how the model had not 
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been implemented in the manner in which it was intended. One principal noted, “Evaluation is 
based on perception rather than observation,” while another commented, “A principal’s job is not 
easily defined in just a couple short observations (and I would venture to say many districts are 
not even completing those).” There were also comments from principals that were interpreted as 
concerns about lack of inter-rater reliability, such as, “It depends on the evaluator,” and “... the 
challenge lies in developing a common understanding of the competencies being measured.” 

Participants also indicated that the RISE model did not adequately represent the 
responsibilities of a principal, including comments implying skepticism in the required value- 
added measures. For example, “It is hard to statistically evaluate all facets of the principal’s 
job.” “The RISE System does not support the management job principals have.” And, “The 
RISE model does little to offer opportunity to react to the daily demands and tasks principals 
must perform that impact the climate and general management of the school.” 

Other themes suggested perceptions that the RISE model was impersonal, too broad in 
scope, and created an unhealthy climate. One superintendent remarked that, “It is not rich in the 
dialogue that is essential for trust and professional growth for principals who are eager to leam 
and improve the culture for learning in their schools.” Another superintendent noted, “It is very 
structured and not relationship-based when implemented as written.” A principal commented, 
“One big challenge from our previous model is the competition it has created between those 
ranked highly effective to those who are ranked effective.” Another principal noted, “One size 
fits all systems may improve efficiency of procedures and human capital decisions, but very 
seldom have long-term effects with positive culture needed to improve performance.” Table 4 
summarizes the themes coded from participants’ comments for challenges of the RISE model. 

Table 4 

Descriptors of Challenges of the RISE Principal Evaluation Model 


Principals _ Superintendents _ Total 


Challenges of RISE 

n 

% 

n 

% 

n 

% 

Too time consuming 

33 

23 

37 

43 

70 

31 

Not representative of the job 

24 

17 

14 

16 

38 

17 

Lack of implementation fidelity 

22 

16 

12 

14 

34 

15 

Too broad and impersonal 

23 

16 

11 

13 

34 

15 

Creates unhealthy climate 

11 

8 

5 

6 

16 

7 

Comments not used 

10 

7 

2 

2 

12 

5 

Did not use other model/no comparison 

11 

8 

1 

1 

12 

5 

No challenges 

7 

5 

4 

5 

11 

5 

Total 

141 

100 

86 

100 

227 

100 


Summary of Results 

This research investigated perceptions of principals and superintendents regarding a new 
principal evaluation model in Indiana. Our purpose was to gain a sense of practitioners’ 
perceptions of model effectiveness, utility, and implementation fidelity, and then compare 
superintendents’ views (the evaluators) with those of principals’ (those evaluated). 
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Analyses of responses revealed several perceived areas of model strength such as clear 
expectations, a useful rubric, a focus on data, and increased communication between principals 
and supervisors. However, challenges were revealed including time-based implementation 
issues, doubts about job representativeness, concerns about implementation fidelity, and 
perceptions that the model did not facilitate overall leadership effectiveness. 

Many participants indicated that their districts had made modifications to the state- 
developed RISE model, including changing the rubric, the metrics, and the recommended 
protocols. Participants’ rankings of the standards-based competencies were not found to be in 
direct alignment with IDOE priorities (the IDOE prioritizes Human Capital Management on the 
principal licensure assessment, but our participants ranked Student Learning as the most 
important principal competency). Several statistically significant differences were found 
between principals’ and superintendents’ perceptions of the RISE model in terms of how 
effectively it supported principal evaluative processes, how well it supported improvement in 
principals’ overall leadership effectiveness, and levels of implementation fidelity. Many 
principals in our study did not perceive the RISE model to be an effective tool for evaluating 
principals and would not recommend the model to their colleagues. In general, superintendents 
viewed the model more favorably than did principals. 

Discussion 

In recent years, there has been impetus towards state-level reform efforts that include the use of 
new educator evaluative processes (Gullickson, 2009; Jacques et al., 2012; New Leaders for New 
Schools, 2010). In 2012, the State of Indiana moved forward with an innovative principal 
evaluation model incorporating several emergent practices. This study gathered and compared 
superintendents’ and principals’ perceptions regarding principal evaluation utilizing this new 
model. We found that overall, principals and superintendents agreed that the RISE model 
supported the improvement of principal leadership. However, comparing the two groups’ mean 
responses revealed significantly higher levels of agreement from superintendents. These results 
substantiated Thomas et al. (2000) findings that there were differences between how principals 
and superintendents viewed principal evaluation in terms of importance and usefulness. 

According to Derrington and Sharratt (2008), the foundation of an effective evaluation is 
detennining the competencies or criteria for assessing performance. Stronge et al. (2013) noted 
that there are a host of variables that affect the principal’s position on a daily basis and it has 
been difficult to detennine which parts of the position are most important. To address this issue, 
we asked participants to rank the six RISE competencies in terms of importance, and then we 
compared their rankings with state-level priorities for principals per the state’s building-level 
licensure assessment. 

Principals and superintendents ranked Student Learning as the most important competency 
for principals, closely followed by Instructional Leadership. However, when we analyzed IDOE 
priorities in terms of the competencies, we found that Human Capital Management was given top 
weight, with Instructional Leadership second. Although the remaining competency weights were 
fairly closely aligned with our practitioners’ rankings, we found it disconcerting that the IDOE 
considered Human Capital Management to be a top priority in principal practice. Our 
participants’ ranking of Student Learning as the most important competency aligns with research 
(e.g. Marzano et al., 2005). Marzano et al. found a compelling relationship between leadership 
and student achievement and that effective leaders align their actions with the priorities in the 
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school. In other words, Marzano’s research team concluded that effective leaders focused on 
student learning make a difference in students’ achievement. We believe that competencies 
closely associated with student achievement should be the top priorities on building-level 
principals’ evaluations. Certainly, human capital management is a responsibility, but may not 
impact student achievement as strongly as a focus on student learning and instructional 
leadership. Therefore, this mismatch in building-level leadership priorities between the IDOE 
and research-based best practice should be addressed. 

In terms of use of the RISE model, we found that approximately 60% of participants’ 
districts used the model as is, while 40% had made modifications. We found these results telling 
because modifications indicate that districts were not satisfied with the model as designed. 
Changing the rubric, the metrics, and the frequency or duration of the observations were the most 
common modifications mentioned. These modifications suggested challenges in implementation 
of the original design, which required districts to make adjustments for the model to be 
workable. 

To effect school change Waters, Marzano, and McNulty (2004) noted that the 
magnitude or order of change processes is important. Some changes are considered first order, 
which imply incremental change that is fairly consistent with prevailing norms and values. For 
many school districts attempting to implement this new model for principal evaluation, the 
changes were second order. In other words, the changes were complex, required new skills and 
knowledge to implement successfully, and may have been a dramatic break from past practice. 
The implications of change can vary widely depending on the norms, values, and perceptions of 
the stakeholders in the school district. Unless appropriate practices and strategies are selected to 
support this new initiative, the changes may not be sustainable and ultimately may not have a 
positive impact on student achievement. 

Fidelity of implementation was another area of concern. Our results suggested similar 
findings to a survey by Duke and Stiggins (1985), which found that principals and 
superintendents disagreed on the thoroughness of evaluations, with superintendents feeling more 
satisfied than principals about the process. In our study, superintendents reported significantly 
higher levels of agreement when asked if the model was being implemented with fidelity. 
Implementation requirements of the model were outlined in the RISE Principal Handbook 
(IDOE, 2012a); however, it appeared from principals’ written comments that there were 
concerns regarding lack of alignment between written procedures and current practices. 
Principals noted that procedures for implementation were not being followed. In addition, 
principals commented that the results of their evaluations depended on the individual evaluator. 
Principals perceived the lack of evidence gathering, lack of time for superintendents to be in the 
buildings, and lack of acknowledgment of the job responsibilities, to be challenges to accurate 
and consistent implementation. Superintendents indicated that the many responsibilities of their 
position have been barriers to consistency, as well as the evaluation process itself being too time 
consuming to be effective. These findings confirmed results by Thomas et al. (2000) who found 
that principal perfonnance is inconsistently measured. 

However, when asked to compare the RISE model with previous evaluation models, 
principals and superintendents indicated that the new model was more thorough and specific, 
increased accountability and objectivity, facilitated communications, and focused more on data. 
Overall, both superintendents and principals perceived the RISE model to be a more effective 
tool to evaluate principals than traditional locally-developed models. Prior to 2012, Indiana 
school districts had autonomy to create their own evaluation model or adopt a pre-existing 
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model. Historically, principal evaluation was not a priority and in some districts, it did not occur 
at all (NAESP, 2012). The RISE model was Indiana’s first attempt to develop a system that 
could be used statewide. Our results suggest that although improvements to the RISE model 
may be needed, the new model is perceived to be more effective than prior models. 

Nevertheless, there is discrepancy between principals and superintendents regarding the 
model. More principals than superintendents indicated a lack of faith in the effectiveness of the 
model to accurately evaluate job performance. One principal stated that the evaluation is based 
on perception rather than objective measures. Another principal commented that the system is 
only as good as the evaluator’s perception of the principal’s work, as the superintendent is 
limited in what they actually observe. Whether it is lack of training or lack of understanding that 
leads to ineffective evaluative processes, these differences in perceptions between principals and 
superintendents are important findings in this study. These results suggest the need for research- 
based revisions to the RISE model in addition to state-led training initiatives. 

Implications and Recommendations for Practice 

Portin et al. (2006) found that principals’ viewed their evaluations as perfunctory and having 
limited value. In Indiana, it is possible that low implementation fidelity and evaluators not 
following the protocols contributed to principals’ perceptions regarding lack of internal 
consistency with their RISE evaluations. Our findings imply that there is still a need for training 
on the RISE model for both principals and superintendents. 

By conducting this study, we were able to gain a clearer understanding of the perceptions 
of principals and superintendents regarding the RISE model for principal evaluation in Indiana. 
Results indicated a clear delineation between superintendents and principals regarding the 
model’s effectiveness to evaluate and support principals in their leadership roles. Although the 
RISE model was perceived generally as an improvement over traditional locally developed 
models, our principal participants did not see the model as providing an effective tool for 
principal evaluation and would not recommend it to colleagues. Superintendents, on the other 
hand, would recommend the model and perceived it to be effective, albeit time-consuming. 

This gap between superintendents’ and principals’ perceptions of the effectiveness of the 
model highlights the need for ongoing support of principal practice. Even though 
superintendents and principals indicated that communication had increased with the 
implementation of RISE, our results indicated a certain lack of confidence among principals in 
the evaluation process that may not be overcome without improved fidelity to the process. 

Although the Indiana RISE model has now been implemented and many districts have 
made modifications, there is still a struggle to implement a system of evaluation that is 
acceptable to all parties and provides desired results in this era of accountability. This study 
revealed a strong need to revise and align the RISE model to more recent collaborative 
leadership behaviors (Clifford & Ross, 2011; Gullickson, 2009, NAESP, 2012; Tran & Bon, 
2015) that may provide supportive and accountable measures for the role of principal. In 
addition, the standards-based priorities of the state must be aligned with current research on best 
practices for principal effectiveness (Marzano et al., 2005; Stronge et al., 2013) in order to focus 
on the competencies that foster meaningful improvements in leadership practices. 

Moreover, the responsibilities of the principalship are multi-faceted (Kemp-Graham, 
2015; Young & Mawhinney, 2012). Both superintendents and principals suggested that some 
indicators of the principal’s role were not addressed in the RISE model. It is important that both 
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superintendents and principals regard the evaluation process as comprehensive and objective in 
order to accurately measure, support, and increase leadership effectiveness. In revising the 
current model or developing a new statewide model, it is important that practitioners have a 
voice in the process. This voice should not be limited to a select few serving on a state- 
appointed development committee. Instead, broad opportunity for input should be facilitated 
through electronic surveying or other means. The NAESP and NASSP research committee 
found that an essential feature of sound practice was that principal evaluation models be created 
by and for principals (Clifford & Ross, 2011). Practitioners in the schools are the field experts, 
and their input provides valuable feedback for improved, yet practical evaluative tools and 
processes. However, in Indiana, principals have been involved minimally in the development 
process and change in this practice is vital to a more transparent and inclusive effort. 

Finally, there is a need for greater collaboration between principals and superintendents 
in developing clear and concise implementation criteria that can be consistently employed to 
attain the highest level of support for principal practice. These efforts could ultimately assist in 
increasing overall consistency in effectiveness of the evaluation process. 

Limitations and Need for Further Research 

A limitation of this study was that it involved educators in only one state, Indiana. Principal 
evaluation research with a broader range of participants across several states would increase our 
understanding of practitioners’ perceptions regarding emergent evaluative practices. 

Another limitation was that because the survey was anonymous, there was no way to 
control for the possibility that more than one central office administrator responded to the survey 
from within a single district. This limitation may be somewhat reduced since many districts in 
Indiana only have one person in the central office responsible for principals’ evaluation. 
Therefore, it is likely that only one superintendent (or the superintendent’s designee) responded 
to the survey within each district. 

An additional limitation was the lack of survey questions specifically addressing 
emergent practices. For example, in hindsight, we wished we had included a separate question 
about participants’ perceptions of the Administrative Student Learning Outcomes and also the 
VAMs incorporated in the RISE model. Several open-ended comments from participants 
suggested low confidence in the VAMs, prompting further inquiry. The RISE model currently 
requires that 50% of the principal’s evaluation be based on a combination of student achievement 
indicators and the school’s letter grade. However, recent research has called into question the 
use of VAMs for evaluation purposes, suggesting that gains in student achievement may be 
influenced by a plethora of factors (AERA, 2015; ASA, 2014; Darling-Hammond et ah, 2012). 
Therefore, it would have been interesting to gather practitioners’ perceptions on this matter as 
this topic necessitates further exploration. 

Furthermore, follow-up studies are needed to explore the local modifications of the RISE 
model, which might uncover potential practices or competencies that would enhance the 
effectiveness of the evaluation system. This might also assist in developing guidelines for 
modifications in order to develop more statewide consistency in evaluation processes. 
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Conclusion 


This study explored perspectives of superintendents and principals regarding the RISE principal 
evaluation model, a state-developed model that was implemented in 2012 in response to 
legislation passed in Indiana mandating reform in educator evaluation. The development of the 
RISE model represented a paradigm shift in the state because school districts previously had 
local control in developing their evaluation models. This research was important to undertake, as 
the Indiana RISE model for principal evaluation had not yet been studied. Through this study, 
we hope to add Indiana practitioners’ voices to the developing professional dialogue about 
effective leadership evaluation and whether or not this new model is meeting intended purposes. 

This research provides insights into the perceptions of currently practicing 
superintendents and principals that potentially could guide revisions to the Indiana RISE model. 
This baseline understanding of the model is important as it informs state-level decision-making 
and also guides future comparative research. This research may also be helpful in other states as 
they review and revise their policies and systems for principal evaluation. Globally, this study 
has significance for educational researchers, practitioners, and policy-makers because it increases 
understanding of evaluation models and strategies used for school leaders, with potential 
recommendations for improving or sustaining practices. The advancement of excellent school 
leadership for all students in today’s society should be on going and requires thoughtful 
examination of practice. 
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